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Summary 


^  The  study  of  dynamic  programming  processes  of  continuous 
type  gives  rise  to  functional  equations  of  the  form  dx/dt  * 
f(x,t;q).  In  tnls  paper  we  present  a  summary  of  some 
basic  results  concerning  the  existence  and  uniqueness  of  solutions 
of  this  equation.  Detailed  results  will  appear  subsequently.  ^  . 
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1.  Introduction. 


In  the  first  paper  of  this  series  *,  c f  also 


we  discussed  the  existence  and  uniqueness  of  solutions  c  the  functional 
equation 

f(P)  -  Max  f(T(P,Q)),  (1.1) 


the  basic  functional  equation  of  the  theory  of  dynamic  programming. 

In  this  paper  we  shall  present  the  first  of  a  series  of  Investigations 
of  various  continuous  analogues  of  the  foregoing  equation.  Our  results 
here  concern  the  nonlinear  system 


-  -  Max  f . (x, t;q) ,  x.  (0)  -  c.  ,  1  -  1.2, ...,n,  (1.2) 

dt  q  1  1  1 

which  we  shall  write  in  the  vector  lorm 


—  -  Max  f(x,t;q),  x(0)  -  c  (1.3) 

dt  q 

The  quantities  x,f  and  c  are  n— dimensional  vectors,  while  q  Is  an 
jia-dlaenslonalj vector  whose  components  may  range  over  a  finite  cr 
denumerable  set,  or  continuum  of  values. 

In  subsequent  papers  we  shall  present  analogous  results  for  partial 
differential  equations  of  parabolic  type,  for  the  first  order  partial 
differential  equations  obtained  from  the  calculus  of  variations 
and  for  other  classes  of  functional  equations  as  well.  A  detailed 
exposition  of  the  results  of  this  note,  and  a  discussion  of  their  relation 
to  an  interesting  class  of  problems  In  the  calculus  of  variations  will 
appear  elsewhere. 

Equations  of  the  above  type  appear  In  the  study  of  continuous 
decision  processes;  some  particular  examples  have  been  treated  in 
unpublished  notes  by  Copeland  and  Darling.  In  many  ways,  these  equations 
constitute  natural  generalizations  of  linear  systems,  and  thus  play  an 
Important  role  in  analysis,  apart  from  their  probabilistic  applications. 


(i) 


Furthermore,  ae  we  shall  see  below,  some  Important  types  of  equations, 
such  as  the  Rlccatl  equation,  can  be  written  In  this  form. 

2.  A  General  Existence  ar.d  Uniqueness  Theorem.  u#  ^  state 


a  g'-neral  res  t. 

Theorem  1.  Consider  the  system  In  (1.2).  Let  us  assume  that  f(x,q,t) 
satisfies  the  following  conditions : 


a.  i |  f (x,t;qj  -  f (y,t;q)  I i  £  k(q,t)  | |  x-y  | j , 

for  x,y,t  In  the  region  R  defined  by  the  Inequalities 
| |  x-c  j  |  £  a,  ||  y-c  ||  £  a,  0  £  t  <  tQ,  for  some  non¬ 
zero  constants  a  and  tQ. 

b.  The  maximum  value  of  f1(x,t;q)  Is  assumed  for  a  function 
^1  "  q^(x,t)  which  Is  uniformly  bounded  In  R. 

c.  The  function  k(q,t)  l_s  uniformly  bounded  for  uniformly 


bounded  q  and  for  t  In  R. 


Then  there  Is  a  unique  solution  of  (1.3)  for  0£t<tj,  where 
t i  l£  some  non-zero  constant  less  than  tQ .  This  solution  may  be 
obtained  as  the  limit  of  the  sequence  defined  by 


Ao  t 

xn+l  "  c  +  Max  f (xn,s;q)ds 

Q 


(2.2) 


There  are  many  Interesting  questions  concerning  the  structure 
of  the  solution  x  and  the  policy  function  q  which  we  do  net  have  space 
to  discuss  here. 

3*  Quasl-Llnear  Systems .  u8uaif  *e  can  obtain  a  much  more  complete 
result  for  systems  of  linear  type. 

Theorem  2.  Consider  the  system  _ , 

(j.i)l 


dx, 


dt 


Max 

Q 


N  i  _ — •  • 

f b. (t;q)  +2  a. .(t;q)x J  ,  x. (0)  -  c.,  1-1,2, ... ,n 
L  1  J-l  w 


Where  we  assume  that 

Maxi  a11(t;q)  I,  Max  |b.(t,q)|  <  f(t)  (3*2) 

q  J  « 

with  f (t)  lntegrable  over  any  finite  t-lnterval . 

Then  there  l£  a  unique  solution  for  t£0  which  may  be  found  by  the 


(2) 


>*»•  atthod  of  successlv  approximations  described  above. 

In  many  cases,  we  can  determine  the  asymptotic  behavior  of  the 
solution,  see  ^  for  a  discussion  of  this  problem  for  difference  equations 
Results  of  similar  type  may  be  established  for  differential-difference 
equations,  and  more  general  types  of  integrc-differentlal  equations. 

**  &BProx*?i?*f0,n  iH  §P5£?. •  The  approximation  technique  used 

above  was  the  classical  one.  Let  us  now  describe  another  type.  Clven 
q^t),  for  each  index  1,  we  •  ay  compute  an  initial  approximation,  x°, 
using  the  equation 

-  A(tjqo)x°  ♦  b(qQ)x,  x°(0)  -  c  (4.1) 


dx 

dt 


To  obtain  a  better  approximation,  we  determine  qi(t),  again  for  each 
1,  by  the  condition  that  q»(t)  maximize  the  expression  ££  j (t;q)x^+ 
*>*(*#<1)  J  •  Having  determined  qi(t)  in  this  way,  we  corapulie^a  second 
approximation  x1  using  (4.1)  with  ^  replaced  by  qt.  We  now  continue 
in  this  fashion,  computing  a  sequence  of  policy  functions  J  j,  and  a 

(  n  )  ^  ' 

sequence  of  approximations  {x  5  .  This  type  of  approximation  we  call 
"approximation  in  policy  space",  see  Concerning  this  process,  we 
can  establish  the  following  result: 

Theorem  5.  I£  the  linear  system, 

gf  -  A(t,q)x  +  f(t),  x(0)  -  0.  (».2) 

has,  for  any  fixed  q,  the  property  that  f  >  0  for  t  >  0  implies  that 
x  £  0  for  t  >  0,  then  approximation  In  policy  gpace  yields  monotone 
convergence  to  tne  solution. 

In  Particular,  there  is  always  convergence  wnen  x  Is  a  scalar 
variable . 

If  “•  (q)  Is  a  constant  matrix  for  each  allowable  q,  then  a  necessary 
and  sufficient  condition  for  the  above  condition  to  hold  Is  a^,>0. 
for  1  d  J;  If  A(t,q)  is  variable,  then  a  sufficient  condition  Is  a^(t) 

£0,  for  1  /  J;  see  '. 

5.  The  Rise  itl  Equation.  An  llT,port£int  tool  in  the  qualitative  and 

quantitative  theory  of  the  second  order  linear  differential  equation 
u"  +  a(t)  u'  +  b(t)u  ■  0  is  the  transformation  of  the  equation  into 

the  form 


(?) 


*•  .  _*»  -«(t)»  -  b(t)\  (5.1) 

the  Rlccati  equation,  by  mesne  of  the  substitution  u  ■  ex^  ( (/*  vd  t ) . 

We  now  observe  that  the  Rlccati  equation  may  be  written  In  the  form 

V  •  Min  V  2v(t)  -«(t)v  — b (t )^J  (5.2) 

<1 

an  equation  of  the  above  general  type.  This  representation  furnishes 
a  new  approach  to  the  study  of  the  solutions  of  (5*1).  There  are  several 
other  classes  of  equations  of  both  analytic  and  physical  Interest  to  which 
this  quasi-linearizaticn  may  be  applied.  TMs  will  be  discussed  elsewhere. 
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